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ABSTRACT 



One of six summaries of workshop sessions (See TM 
000 130), designed to strengthen the evaluation of cosily urograms 
and their effects, this handbook presents an analysis of both random 
and nonrandom sampling errors by application of the Bayesian model. 
This model attempts to formalire the process and procedures o f 
inference from data through explicit consideration of the 
researcher's beliefs, insofar as they beat upon the study underway, 
^hus, if he feels that nonresponse error, for example, is likely to 
be large, this feeling is incorporated into the model. Major error 
sources for each parameter are sampled and values of all the 
variables ar° determined to show how these parameters define the 
sampling problem. Another method involves formulation of error ratios 
in torus of certain established values. An examnle of the application 
of the model to a survey using mailed questionnaires is given. (tP) 
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FOREWORD 



The increased competition for the tax dollar has caused and will continue 
to cause more rigorous evaluations in all fields of education, particularly at 
the Federal level. Increasingly, legislators and their constituent taxpayers 
are demanding hard data which will Indicate whether a costly program is achieving 
that which it has purported to achieve. Under these conditions, evaluation at 
all levels must satisfy the criteria elements of significance, credibility, 
and timeliness, Within this framework evaluative techniques must be strengthened, 

Appropriate departmental personnel believed that strengthening the 
evaluative effort of the State might start with the Elementary and Secondary 
Education Act (ESEA) in general and Title III of that Act in particular. Further, 
it was believed that the 16 existing Regional Centers contained evaluators who 
might be in a strategic position to disseminate Information gained through a 
workshop approach to the problem on the State level. 

Leo D. Doherty, Supervisor of Education Research, of the Division of 
Evaluation, was asked to organize some review and training sessions appropriate 
for the task. He selected people from within the State to prepare and conduct 
formal lessons accompanied by simulated experiences and related materials. This 
document is one in a series of six summaries of sessions completed in a 3-roonth 
period terminating in February 1969. 

While the sessions were paid for out of Title HI funds, the contents 
are appropriate for use with other Titles such as 1, or other large program 
evaluative problems such as those encountered in N.D.E.A. Title 111, Urban 
Education, or the like. 

This document on Sampling Model was prepared by Donald Heyer, Syracuse 
University. 
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SAMPLING MODEL 



Analysis of Errors in Sampling 



Two basic kinds of errors: 

1. Random sampling errors 

2. Nonrandom sampling errors 
Usual procedure: 

Take a sample. Obtain a value for the relevant statistic. Use this 
as a point estimate of the population value and/or construct a con- 
fidence interval. 

Example: Suppose we want to estimate proportion of parents favoring a 

particular kind of reorganization of the schools. If we define the 
family as the sampling unit, we take a random sample from some list 
of families and ask then, the question. Observed statistics are 
N *= number of families in sample and p ** proportion saying they are 
favorable. 

Let fl be the population proportion. From theory we know that the ciean and 
variance of the sample proportion is: 



Therefore, a point estimate of x is p and a large sample confidence Interval 



where * a /2 * 8 °PP er c/2 point of the normal distribution. The difficulty 
is that the above confidence interval considers only the random sampling error. 
If the nonrandom sampling errors are large, such as the error from nonresponse, 
then our belief that the obtained confidence interval Includes the true value 



E(p) ** * 

V<P) - n(l^)/N 



for * is: 
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could be quite small. What is required is a model which would enable us to 
also consider nonrandom sampling errors and Incorporate any knowledge we have 
about them Into our analysis. The advantage of doing this Is twofold: 

1. Our final reported Interval would be a 
more adequate statement of reality; 

2. To aid us in the design of future studies. 

In connection with this last point note that the usual methods give little 
guidance as to design. The preferred study is the one with the larger sample 
size if we only use the length of the confidence interval as the criterion. 
Implicitly, the assumption is that studies are always performed "properly" and 
the sample is random. In actual practice this is seldom the case. 

The Bayesian model, using subjective probability as a cornerstone. Is an 
attempt to formalize the process and procedures of Inference from data with 
explicit consideration of the beliefs of the researcher, Insofar as they bear 
on the present study. If the researcher thinks that nonresponse error, for 
example, la likely to be large, then he incorporates these feelings Into the 
model. The choice of design will then be made conditional on this concern. Of 
course, good researchers have always done this In an informal way. The only 
difference Is that the Bayesian model makes it a formal process. The advantage 
Is that all decisions of any kind are now explicit, and one can "research" the 
model Itself for questions such as sensitivity, Inter-relationships of decisions, 
etc. 
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Major Error Sources (Sampling) 



1 . 



2 . 



3 . 



4 . 



5 . 



Frame error 

This source of error may exist If the frame from which the deslreo 
8 ample Is drawn does not include all of the target population, 
e.g. If one used the telephone book as the frame 

for residents in a city, then those not owning 
telephones are automatically excluded. 

Selection error 

This error may occur if certain elements in the frame have a 
greater chance of being included in the sample than others. 

(This is sometimes dealt with by the use of weighting factors.) 
e.g. Some people's names may appear on the frame list 
more than once as, for example, parents with 
more than one child in school. 

Nonresponse error 

This error occurs if responses are not obtained ftom all the unito 
in the planned sample and if those not responding differ on the 
variable under study from those responding. (It may be convenient 
to decompose this error into refusals and noncontacts.) 

Measurement error 

This error occurs if we do not obtain accurate answers. This can 
depend on the way the question is asked, the order of presentation, 
interviewer variables, etc. 

Random sampling errors 

This is the error usually assessed by use of the standard statls* 



tical formulae 
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Specification of Parameters 

Tfy - proportion in population with attribute, y. 

7i£d[ y = proportion of those having y included in defined frame . 
n fa| y “ probability of selecting a unit with y in the sampled frame. 
n rl f ,y ** proportion of those in sampled frame having y who would 
respond if sampled. 

n s|r y “ proportion of those responding who have y and would say 
they have y. 

All subscripts with "bars" over them mean the negation, e.g. 

« proportion of those who have y and would not respond . 

Sample Data 



r 

islr 



B proportion of planned sample who respond. 

“ proportion of sample members who say they have y among those 
who respond. (This is the usuil statistic reported.) 



0 
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If the values of these parameters » together with the corresponding para* 

meters for the not * y's, are known, then the model for the sampling design is 

completely specified. 

To elaborate on these concepts, we will perform a population analysis 
assuming we know the values of the parameters. In practice we will be uncertain 
as to the exact values of the parameters, but using our "best" guesses can be 
helpful c 'nee the implications for the sampling design will become clear. If a 
large bias is Indicated, then even though we may be uncertain as to the actual 
magnitude of the bias, we may be confident about its direction, and wc may be 
motivated to think about alternative designs. This will serve as a preliminary 
step for a more formal analysis. 

Suppose we are interested in the proportion of families who are positive 
with respect to a certain attitude. (The head of the household defines the 

family for the purpose of the study.) We will sample using a list of residents 

from the city directory which is almost one year old, and we will send a 
questionnaire to those falling in our random sample, let us assume that from 
our past knowledge wr think that those vho are positive towards the issue arc 
also those who are more mobile in terms of residence, and therefore, are more 
likely to be missed by using a city directory this old. 

Tne tree diagram shown in Figure 1 is one possible representation of the 
population and associated sampling design. Starting from point (a) the two 
branches show the values *y and *jr to be .8 and .2, respectively. That is, our 
best, estimate of the proportion in the population who are positive is ,8 and the 
estimate of the proportion negative is, therefore, .2. Itie branches at point's 
(b) and (c) show the sources of frame error. By using the directory, ie miss 
some of the members of the population. The tree shows that our estimate of the 
proportion of those fivorable (saying "yes") vho are Included in the frame 




Representation of Population Composition 
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(directory) is .9 or " «9. Then,, Similarly, the 

proportion of those unfavorable (saying "no") included in the frame is .95 
or ir | „ * .95. Then, ir-jrjj — " .05. Because of our feelings that those 

favorable are more mobile, and therefore, would have more likelihood of 
being missed, we have set "fdjy > ^ f d | y * 

The branches at points (d) c id (e) show the sources of selection error. 

We may send the questionnaires to the addresses listed in the directory, but 
they may not be delivered to the sampling units because of "no forwarding address" 
etc. One could lump this type of selection error with frame error, but it 
may be convenient to keep them separate for this problem. Similarly, if we 
were calling the sample units on the telephone and we couldn't reach them, we 
would find it convenient to think of this as a source of selection error. The 
difference is that frame error refers to the possession or nonpossession of the 
unit's name, and selection error refers to the likelihood of the unit to be 
given a chance to respond. The figure shows that selection error is assessed to 
be small since i = and "fa^y = ’ ^ while Ti£ a |y = -l. That is, no 

selection error for those who are unfavorable. 

The branches at points (f) and (g) show the sources of nonresponse error. 
This refers to the nonreturning of the mailed questionnaires or the refusal to 
respond over the telephone. For this example we are assuming that those who 
are favorable will be less likely to return the questionnaire than those unfavorable 
• might be derived from a general theory that says the favorable group is more 
active in community affairs, busier, etc. and therefore, more likely to ignore 
our request to return the questionnaire. The unfavorable group may be more 
interested in making their views known, etc. Consequently, “ .8 and 

^rj f y ~ *2 or the proportion of those favorable, and who are contacted, who 

will respond is .8 while the proportion not responding is .2. For the unfavorable 
group , it | _ = .9 and tt . _ = . 1 . 
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Finally, points (h) tnd (i) show the sources of measurement error. Again 
from the "busy" hypothesis, the favorable group may be more likely to misread 
the question and say "no" when their true attitude is "yes". Or it may be that 
some think It socially unacceptable to Indicate a favorable attitude so they 
deliberately lie. For this problem, ^slr^y® .95 and 7T s'|r,y = .05 or 95 percent 
of those favorable and who return their questionnaires say they are favorable 
while 5 percent say they are unfavorable. Similarly, "sir,/ = .98 and 

ir s‘|r»y ^ .02. 

By multiplying the numbers along any path to an end-point, we obtain the 
percentages for all possible events. The product of the numbers on the top 
path is .542 which means that, apart from random sampling error, 54.2 percent of 

the total population would be people who are favorable and would be so classified 

\ 

in the Sample. Another .3 percent would be classified as favorable because of 
"unfavorable" people changing their response. The total percentage of "yesses" 
in the sample is then, ,545. Similarly, the total percentage of those indicating 
"no" would be .197. This would mean that of the .742 responding (.545 + .197), 
the percentage of favorable is 73 percent. Since the true value is .8, we 
would have about a 9 percent underestimate. 

The chart also shows that 9 percent of the population are not included 
in the defined frame and that .007 is the likelihood of a "favorable person" not 
being included in the actual sampled frame (selection error). The nonresponse 
rate is 16.2 percent. If the planned sample size is larger than about 150, then 
the obtained sample would be larger than 125 and the .95 confidence interval 
shown earlier would be, on the average, less than seven percentage points about 
.73, so that the interval would fail to cover the true value of .80! 

We are not claiming that one woulr actually know these parameters as 
precisely as we have indicated on the chart. Our purpose is to show how these 
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parameters define the sampling problem. The five parameters on the top branch 
and the four on the branch emanating from point (c) completely specify the 
population. 

If a joint prior distribution coulc! be assigned to these nine parameters, 
then the sample data consisting of the proportion of "yesses", ^s|r, and the 

A 

response rate, ^r, could be combined with the prior using Bayes 1 theorem to 
obtain the joint posterior distribution. The marginal posterior distribution of 
could then be obtained by summing over the eight other parameters. This 
marginal would summarize our current 3tate of knowledge about v y and point 
and/or Interval estimates could be obtained If desired. 
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Formulation in Terms of Error Ratios 

The assessment of the joint prior distribution of the nine parameters is 

difficult partially due to dependencies which may (probably) exist among them. 

One approach which has been suggested is to formulate new parameters which are 

1 / 

ratios of the original nine (or functions of them). It is hoped that the 



dependencies are, thereby, eliminated. Error ratio model: 

* 



IT 

= 

s [r 



VI fd 

(a) 



Vlfd . n y|fa . V 
Vlfa % % |r 

(b) (c) (d) 



ft I 

-ilk 

*s |r 
(e) 



The 1< T t hand side is the ratio of the true proportion of "yesses'' in 
the population and the observed proportion of those responding who say "yes" 

(or are favorable). 

Frame Error 

Ratio (a) is a measure of frame error since it is the ratio of the true 
proportion in the population and the proportion of "yesses" in the defined frame. 
If its value is one, there is no frame error while a value greater than one would 
Indicate that the proportion of "yesses" in the frame is less than that in the 
population. A value lead than one would Indicate the reverse, n yl fd is related 
to ^fdly used in the tree diagram by: 



M fd - 



V fd[ y 



yfd|y + yfd|y 



7T TT 



Using the numbers from the example: 



"yl fd = (-8)(.9) _ 



(.8)(.9) + (.2) (.95) 



■ 72 
.91 



= .791 



1/ R.V # Brown, ‘'Credence Analysis: The Appraisal of Research Error, 11 
unpublished doctoral dissertation. Harvard University, 1968. 
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This means that of the total population, 91 percent are Included In the frame. 
Of this group 79.1 percent are "people who are favorable." The frame error 
ratio is: 



J0_ 

.791 



1.0101 



In thinking about frame error it may be convenient to think of the 
ratio of proportion of "yesses" in the "uncovered" population to the pro- 
portion of "yesses" in the covered population since: 



"y|fd 



_ n_ ( hll± 



fd 



Vlfd 



TS 



where it-— is the proportion of population not included in the frame and 
is the proportion included in the frame. Using the figures in the 

tree : 



(.09) 



.08/ 

' .09 



•7 



91 



, .8888 , 

- - 91 - <- 09 > "TmT J + - 91 



= (.09) (1. 12) + .91 = 1.0101 



With 12 percent more "yesses" in the uncovered portion and 91 percent of 
everybody included, it is seen again that frame error is 1.0101. 



Selection Error 

Ratio (b) measures selection error. It is the ratio of the proportion 
of y’s included in the frame to the proportion of y's in the actual sampled 
frame (or probability of selecting a "y" in the frame). 

The parameter, * y |f a> is related to ^fal y , f d use ' ! ln the tree by: 

„ | {a _ n y rcfdjy n fa| y , fd 

^ iTy iTfd|y ^faly.fd + 11 y ^f d | y" ^faly, fd 

ff y| fd ^faly.fd 

’VI fd ’'faly.fd + Vi fd "faly.fd 
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For our example: 

«,]- = ( 1 791) ( . 99) = ,783 = 

^' fa ( . 791) ( . 99) + (.209) (1) .992 ‘ /<5 . 

Selection error is “ 1.002 

• 7o 7j 

Random Sampling Error 

Term (c) measures random sampling error. It is the ratio of the 
proportion of "yesses' 1 in the actual sampled frame to the true average value 
in the planned sample. For most situations we will argue that the expected 
value of this ratio is one. 



Nonresponse Error 

Term (d) measures nonresponse error. It is the ratio of the true 
average value in the planned sample to the true average value in the achieved 
sample. 

7 T i is related to 7r i by: 
y|r r\y 

* y | r „ jj My 

MfaMy^yjfaMy 

For our example: 



y| r 



( . 7895) ( . 8) 



(.7895)(.8) + ( . 2105) ( . 9) 
,6316 



,8211 



7692 



In our example, there was no random sampling error so we will use 



*y = "y|fa 



,7895 



.7895 



The nonresponse error ratio is: * = 1.0264 

• / 07 Z 

It may be helpful to the assessor to think of the ratio, *y|-f/^y| r 
which Is the ratio of proportion of "yesses" In the nonresponse group to the 
proportion of "yesses" in the response group and use: 
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For the example. 





.179 



• 143 / 



162 



.769 



(.821) 




.821 



1.0267 



Measurement Error 

Ratio (e) measures measurement error. It is the rstlo of the true 
average value In the achieved sample to the obtained value. That is, fi g | r 
Is the act '.1 proportion of those responding who say they are "yes." In 
the planning phase, this would be the expected value to be obtained. 

For our example, if i = ,73 



or 




.769 

.73 



1.053 



Total Decomposition 



For the example the model Is decomposed as: 




1.10 = ( 1. 0101) (1. 002) (1)(1. 026) (1. 053) = 1.12 (round-off error) 

This Is a case where all error ratios were greater than one which 
would result In the underestimate, .73. It Is possible. In general, for 
some of the ratios to "cancel" each other resulting In no net error. 
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Example of Use of Model 

2 / 

R. V. Brown — has given an example of a survey using mailed question 
naires. The problem was to estimate the demand for parking space if meters 
were introduced in the town of Camford, England. 

The sampling frame consisted of a list of 10,000 owners of registered 
cars. Questionnaires were mailed to 1000 owners requesting them to show on 
bar charts the hours between 8 A.M, and 6 P.M. on three specific days they 
would be parked if ample metered space were available. Nine hundred usable 
replies were received and 90 said they would be parked at peak hours. 



The error rate decomposition is: 




where T = total spaces required 

ft g | r = proportion of respondents who say they need space 
N = 10,000 

Note that the left hand side is a ratio of numbers of spaces, not proportions 
as before. 

For each ratio we assess the expected value and variance of our 
distribution of belief concerning the ratio. However, since it is difficult 
to assess the variance we will use the 95 percent credible interval of the 
distribution. If the natural logarithm of the ratio is normally distributed, 
then the range of the .95 C.I. is approximately four standard deviations. 
Furthermore, it is convenient to use the rel - variance which is the variance 
divided by the square of the mean. 

Then, 

rel-variance = v * 

2 7 R. V. Brown, "Evaluation of Total Survey Error", Journal of Marketing 
Research , Vol. IV, May, 1967. 
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Assessment of Frame Error 

iyN7Fy| Is the ratio of total spaces required to the number required 
by the 10,000 owners on the list. All potential parkers are not included in 
the frame since out-of-town parkers will use some space. 

The assessor thinks total demand will be about 1.1 of that demanded 
by in- town parkers. Out-of-town parkers will use about 10 percent of the 
spaces required by in-town parkers, so the mean is assessed at 1.1. The .95 
C.l. ranges from .02 to .24. 




(a) 



The distribution is positively skewed since it is quite unlikely for (a) 
to be less than one. Note that the upper limit of the .95 credible interval 
specifies some probability that out-of-town parkers could possibly use 24 



percent or more of the spaces used by in-town parkers. 

*» 2 

= .0025 



r .22 ~ ] 2 r 

= 4(1.1) = .05 



or 



E((a)) = 1.1, v = .003. 



Assessment of Selection Error : 

it is thought that no selection error has occurred. Therefore, the 
expected value of the selection error ratio is one with rel-varlance of zero. 

Measurement Error 

ft I 

In assessing A Xl. r — the assessor knows people are unreliable in what 

Mr 

they say about hypothetical future actions. He feels people will tend to 
underestimate their parking needs. He sets the expected value at 1.2 which 
means he feels that the true percentage is 20 percent higher than that 
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reported. He is somewhat vague, however, so he sets the *95 credible 
interval to be .4 to 2. His prior distribution appears as follows: (Exact 

shape is not critical.) 




Nonresponse Error 

In assessing , , the assessor feels that ihose not responding 

7 y| * 

would be people not especially interested in parking in Caraford. Therefore, 
the "parking rate" for nonresponders would be less than that for the 
responders . 

However, he knows that 



ft yfr 



ft I- 



M 



and since it = .90 and ft i 

r y| r 



- 1 -^ 7 1 + *- 

10, then 



.10 



10 



fUs- 



id 



.10 



1+ .90 « ft y | 7 + .90 



which raean3 ^/fiy| r i *90 since jr — 0. If fty|p B 

Vvit - •” 



.07, then 
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The distribution of the ratio Is, therefore, assessed as shown belcw: 




(c) 






The Ej 

to 1.0. The rel -variance, v 



,97 with the .95 credibly Interval ranging from .95 



[*] 



.0002 



Sampling Error 

The ratio, n y| t&j fty , is the ratio of the true proportion of "yesses" 
in the sampled frame to the true average value In the planned sample. The 
ratio would differ from unity only because of sampling error. 

From theory for binomial sampling: 

E(*y|fa/# v ) - 1 



v(’yl £a /* y ) 



E( TT X 



fa 



) - 1 



n 



Also, If the ratios are independent. 

E <V|fa> * E( V ’ E <*s|r> * E < #yl 7«s| r> 



ft I was observed to be .10, so 
s| r 

E(ft y ) = (.10)(. 97)<1.2) = .116 

1 

v( ft y| fa y4 y ) “ “*4ooo ■ - 0076 

Note that we use the planned sample size of n - 1000 since the nonresponse 
error, resulting in the actual sample of 900, has been separately expressed. 
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Total Error Analysis 



If the error ratios in the model decomposition are independent. 



then 






= E 



s r 



and the rel-variance, v 




Ml 8 . 









s r 



, is the sum of the rel-variances 



of the error ratios plus a product correction term consisting of all 



products two at 


a time 


and three at a 


time. The correction term will 


be small if all 


or all 


but one of the 


rel-variances are 


small. 






Summary Table 




Source of Error 




ratio 


mean 


rel-variance 


random 




fa /. y 


i 


.0076 


nonresponse 




Vvu 


.97 


.0002 


measurement 




,,|r Ait 


1.2 


.1111 


frame 




T / N VI £a 


1.1 


.0025 | 










1 

.0011 


Total 




T / N *s|t 


1.28 


.1225 



correction 



Since N ■ 10,000 and $ . * .10, then the expected number of spaces 

s|r 

T, is: 

£(T) - ( 10,000) { . 10) (1.28) “ 1280 

2 

with a rel-variance of .1225. The variance is V(T) * (1280) (.1225) 

« 200704 and rV(T) * 448. 

The .95 credit 1 o interval is: 

C.I. - 4(448) • 1792 

Thu distribution is not symetrical since this would assign probability 
to negative values of T. We will assume that the ratio, T /« *.ir • is 
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distributed log-normal which would follow if the component ratios were 
log-normal (or numerous by the central limit theorem). 

If this is so, then a relationship between the .95 credible interval 
and the mean is: 



upper limit m mean 

mean lower limit 



Then: 



y. 

1280 



1280 

Jt 



p - * - 1792 

These two equations give p « 2459 and i 

This means that 



667. 



P(667 4 T 4 2459) < .95 with an expected value of 1280 spaces. 

The usual .95 confidence interval for it would be: 



.10 - 2 



/( .10) ( . 90) A. 10) (-90) 

/ 900 < „ y < .10 + 2 / 900 

C £.08 < n y < .12j « .95 



.95 



or in terms of number of spaces: 



[« 



C 800 < T < 1200 



'] 



.95 



Note that the .95 credible interval using error ratio analysis is 
more than four times as large as the .95 confidence interval and the 
expected value of 1280 spaces is 28 percent higher than the point estimate, 1000. 




